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REMARKS 



Claims 1-5 remain pending iii this application. Claims 1-5 are rejected. 
Claim 1 is amended herein to clarify tlie invention and to correct typographical 
errors. 

Claims 1-5 have been rejecied under 35 U.S.C. § 112, first paragraph, as 
failing to comply with the written description requirement. The Office Action 
states that new matter has been added to claim 1 , part e) with the addition of "^and 
the time in months." The recitation that was deemed new matter has been 
removed. 

The OfBce Action states that new matter has been added to claim 1 , part t) 
by amending the claim to calculate the greatest similarity by finding the maximum 
of a range of values of Aj. Claim 1 has been changed to finding the minimum 
value, as had been recited by the claim previously. Accordingly, Applicant 
respectfully requests that the rejection under 35 U.S.C. § 112, first paragraph 
pertaining to written description, be withdrawn. 

The specification has been objected to under 35 U.S.C. § 132(a) as having 
been amended to incorporate new matter by amending the specification to calculate 
die greatest similarity by finding the maximum of a range of values of 
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Aj, The specification has been amended to finding the mimniuin value, as had been 
reciced by the specification previously. Accordingly, Applicant respectfully 
requeiilii ihai the objection to die specification be withdrawn. 

Claims 1-5 have been rejected under 35 U.S.C. § 1 12, first paragraph, as 
failing to comply with the enablement requirement. The Office Action states ihat 
llie recitation in claim 1, line 11 of ""measuring, at one or more points in rime" is 
not enabling since ir does not disclose essential elements. The Office Action is 
requesting the minimum number of points in time for measiuring values and the 
time interval for such measurement. Also, the Office Action states that the 
specification on page K second paragraph, di.scloses that osteodensitometry 
requires that bone density values must be measured at three different times and that 
sufficiendy large intervals between the times of measuring the bone density should 
be selected. As can be seen from claim lb), the equation for M* shows that 
normalization is taking place. Someone of ordinary skill in the art would know 
that there must be at least two points in time since the normalization is done with 
respect to the first point in time, as shown in claim lb). Claim lb) has therefore 
been amended to recite "measuring, at two or more points in time." As for the 
specification's reference to at least three measurements, this is a background 
discussion and does not necessarily teach that at least three measurements are 
needed in die process of die current invention. 

1 S f/Ul ami r«ap lo Eba of 10 Id-OS {rC22 \ wpi 
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As for time intervals, there is no set duration for them. The specilicaEion 
on page 2, tirst full paragraph states that the method of the current invention strives 
to shonen the period for interpreting the interaction of bone destruction processes 
and bone formation processes. Thus, the method of the current invention seeks to 
reduce die measurement intervals. Tn the event that minor or no differences are 
determined during a time interval, one of skill in the art will be able to interpret 
whether these differences are caused by the time intervals selected (such as being 
too short) or whether ihe results simply show a good result of the evaluation 
regarding bone losses. Thus, no set interval is necessary since the user of ihe 
process will determine the interval. Accordingly, Applicant respectfully requests 
diat the rejection imder 35 U.S.C. § 112, first paragraph^ pertaining to enablement 
be witlidrawn. 

Clauns 1-5 have been rejected under 35 U.S.C. § 112, second paragraph, 
as being indefinite. 

The Office Acdon points out the recitation in claim 1 that "K** is the 
number of bone markers. The Office Acdon states diat diis is not a sufticiendy 
specific definition for "K". For example, the Examiner wanra to know whedier 
**K" refers to the bone markers iii a population of patients or the bone markers for 
a single patient. 
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The invention does not seek to make particular specification's for the 
selection of bone markers. Instead, the invention seeks to evaluate the data that is 
available as besi as possible. It is a basic principle of multi-parametric 
classiticaiion to be able to increase the difference between two situations by 
increasing the number of distinguishing features. The data available is what the 
user of the method has measured on the basis of experience and/or possibilities and 
this is the data that is evaluated. 

The Office Action states that the recitation in claim 1, line 11 of 
measuring, at one or more poinis in time"* and the remainder of the claim suggests 
that the "nteasuring, at one or more points hi tune" corresponds to time values in 
the formulae of the processing parts a) - e) but tliat the claim must recite this 
clearly so that assumptions are not required. The claim has been amended 
accordingly. 

The Office Action requested that in claun 1, part b), the recitation of ''a 
table of measured values" be clarified. Claim 1, part b) has been claritied 
accordingly. 

Claim 1, pan c) recites that W ,...Wfc are weighting factors and that in a 
standard determination = 1. The Examiner says that ii is unclear what 
W|...Wk are and has requested clarification. It is the intention of the invention to 
allow a user of the process to establish diese factors and one of ordinary skill in the 
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an would know what these factors are. In panicular, the Minkowski dLsiances is 
somcdiiiifi that one of ordinary skill in the art would be familiar wiih. Although 
not necessarily related to the subject macter of ihe current application, please find 
transmitted herewidi ^DPF-A Perceptual Distance Function for Image Retrieval" 
by Beiiao Li, Edward Chang, and Ching-Tung Wu, which contains some 
information on the Minkowski distances for the Examiner's perusal. 

The Office Action states that claim 1 , part g) recites that Rj(t) are reference 
functions but that the claim does not provide set values for the reference fiinctions 
nor what influences the reference functions. The Office Action then requested a 
dellnicion for the reference hinctlons. Claim 1 refers to the possibility of 
simulating reference values madiematically or of using real values. Tliese 
reference values are associated with the reference functions Rj(t) and arc 
[imc-dependent data. 

The Examiner states that there is no antecedent basis for the recitation of 
''the fir^t and second similari^ dimensions" in claim 1, part f). The claim has 
been amended accordingly. 

The USPTO is hereby authorized to charge any fee(s) or fee(s) deficiency 
or credit any excess payment to Deposit Account No. 10-1250. 
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In light of ihe foregoing, the application is now believed to be in proper 

form for allowance of all claims and noiice lo that effect is earnestly soUciied. 

Respectfully submitted, 
JORDAN AND HAMBURG m.p 



By r./,^^^p Hil^L/^^ 
Bruce 



/Jy and. 



Bruce Hamburg 
Reg. No. 22,389 
Attorney for Applicants 



Ricardo Unikcl 
Reg. No. 52,309 
Attorney for Applicants 



Jordan and Hamburg llp 
122 East 42nd Street 
New York, New York 10168 
(212) 986-2340 

Enc: '^DPF-A Perceptual Distance Function for Image Retrieval" by Beitao Li, 
Edward Chang, and Qiing-Tung Wu 
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DPF — A Perceptual Distance Function for Image Retrieval 

Beitao Li, Edward Change Ching-Iling Wu 

Electrical & Computer Engineering, U.C. Santa Barbara 
beitao@cngineering.iicsb.edii, echang@ece.ucsb.cdu 



Abstract 

Fur ulniost a decade, Content-Based Image Retrieval has 
been iui active research area, yet one fundamental proh- 
lem remains largely unsolved: how to nieasiire percep- 
luai siniiiariLy. To measure perceptual similarity, mosT re- 
Hcaruhurs employ the Minkowski-type metric. Our exten- 
sive dala-mining experiments on visual data show that, im- 
fortuiiacely, the Minkowski metric in not very effective in 
modeling perceptual similahcy. Our experiments also show 
ihat the traditional "litatic" feature weighting approaches 
are not sufficient for retrieving various similar images. In 
ihifj paper» report our discovery of a perceptual distance 
functuni through mining a large set of visual dam» We call 
the diacoverod function dynamic partial distance fiinction 
(DPF). When we empirically compare DPF to Minkowski- 
typc distance flmctions, DPI* pcribrms significantly better 
m finding similar images. The efieetiveness of DPF can be 
well explained by ximilarity Theories in cognitive psychol- 
ogy. 

Keywords: euntent-bascd image retrieval, data mining, per- 
ceptual distance function, similarity search. 

1 Introduction 

Research in content-based hnag^ retrieval has steadily gained 
momentiun in recent years as a result of the dramatic increase 
in the volume of digital images. To achieve effective retrieval^ 
an hna(;e system must be able to accurately characterize and 
quantity perceptual similarity. However, a fundamental chal- 
lenge how to measure perceptual 5nmilarity — remains 
largely unanswered. Various distance functions, such as the 
Minko-^^ski metric, earth mover distance [5], and Ju2zy logic, 
have bc;:n used to mca.surc 5?imilarity between feamre vectors 
representing images. Unfortunately, our experiments show thai 
ihey frequently overlook obviously similar images and hence 
are not (Ulequatc for measuring perceptual similarity. 

guanrifying perceptual similarity is a difficult problem. In- 
deed, may be decades away from fully understanding how 
human perception works. In this project, we mine visual 
data extensively to nfyerse-engineer a good percepmal distance 
iuncdon for measuring image similarity. Our mining hypothe- 
sis is Uiis: Suppose most of the similar images can be clustered 
in a featjre space. We can then claim with high confidence that 
0 the feature space can adequately capture visual perception. 



and 2) the distance ftmctlon used for clustering images in that 
feature space can accurately model perceptual similarity. 

We perform our mining operation in two stages. In the finit 
stage» wc isolate the distance function factor (we use the Eu- 
clidean distance) to find a reasonable feature set. In tiie second 
stage, we freeze the features to discover a perceptual distance 
ftmcdon that can better du^er similar images in the feature 
space. In other words, our goal is to find a function that can 
keep similar images close together in the feature space, and at 
the same time, keep dissimilar images away. We call the dis- 
covered function dynamic partial distance function (DPF). Wc 
en^iiically compare DPF to Minkowski-type distance func- 
tions and show that DPF performs remaiicably better. 

Briefly, the contributions of this paper are as follows: 

« We construct a mining dataset to And a feature set that can 
adequately represent images. In that feature space, we find 
distinct pattenis of similar and dissimilar images, which lead 
CO the discovery of DPF. 

• Through empirical smdy, we demonstrate that DPF is very 
effective in finding images tliat have been transfomied by ro- 
tation^ scaling, downsampling, and cropping, as well as im- 
ages d)at are perceptually similar to the query image (e.g., 
images belonging to the same video shot). Our testbed shows 
that DPF outpcrfomu Minkowski-typc functions by 25 per- 
centiles in recall. 

2 Discoveriug DPF 

To cmsure that sound inferences can be drawn from our min- 
ing results, we caiefiilly construct the training datasct. First, 
wc prepare for a datasct that is comprehensive enough to covcr 
a diveraified set of images. To achieve this goal, we collect 
60,000 JPEG images from Corel CDs and from the Internet. 
Second, we define "similarity" in a slighdy restrictive way so 
that individuals* subjecuvity can be safely excluded. (We ad- 
dress the problem of learning subjective perception in [1. 6].) 
For each image in the 60, OOO-image set, we perform 24 trans- 
formations including scaling, downsamping, cropping, rota- 
tion, and format transformation. (Details of diese uansforma- 
tions are explained in the extended version of this paper [4].) 
The total number of images in the testbed is 1.5 million. 

Our experimental results (see Section 3) show that die per- 
cepmal distance function discovered during the mining process 
on this training datasct, which has a slightly restrictive defini- 
tion of similarity^ can be used effectively to find other percep- 
tually similar images. In other vvords, our testbed consists of 
a reasonable representation of similar images, and the mining 
results (i.e., training results) can be generalized to testing data 
consisting of perceptually similar images produced by other 
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Fig\ire I : The Distributions of Feature Distances. 



meThods ie.g., chawgiog c<miera parameters). 

From each image, we extract 144 features including color, 
texture, and shape as its representation. We discuss what these 
feaiureri are, ami why they are chosen in [4]. In the remainder 
of this section, we focus on examining the Minkowski metric 
and its &mily. We explain why these flmctions are ineffective 
for measuring image similarity, and present our DPF solution. 

2.1 Minkowski Metric and Its Limitations 

The Miakowslu metiic is widely used for measuring similaricy 
between objects (e.g., images). Suppose two objects X and Y 
are represented by two p diniensional vectors {o:x , 353, • - ' , Xp) 
and (i/i* )V2> • ■ ■ * J/p), respectively. The Minkowski metric 
c2(;!r,y)isddinedas 

diX,Y)»(f^\xi-yinK (1) 

i»t 

where r is the Minkowski factor for the norm. Particularly, 
when r is sol as 2» it is the well known Euclidean distance; 
when r is 1> it is the Manhattan distance (or Li distance). An 
objeci located a smnller distance from a quciy object is duemed 
more similar to the query object. Measuring similarity by the 
Minkow.ski metric is bosed on one assuniplion: the similar ob- 
jects should be close to the queiy object iu all dimensions. 

A variant of the Minkowski function, the weighted 
Minkowski distance function, has also been applied to mea- 
sure unage similarity. The basic idea is to introduce weight- 
ing to identify important features. By assigning each feature a 
weighting coefficient flij^ (i = 1 . . p), the weighted Minkowski 
distance function is defined as 



<WA-,y-) = (X]«'<k<-v.r)'. 



(2) 



By applying a sltiiic weighting vector for mcasining simi- 
larity. The weighted Minkowski distance ftmcdon assumes that 
similar images resemble the query imnge(s) in the same fea- 
aires. For example, the weighted Minkowski function impiic- 
irly assumes that the important features for finding a scaled im- 
age arc ihc same as the important features for finding a cropped 
image. 

We can summarize the assumptions of the Minkowski met- 
ric as fofkiws: 

« Minkowski function: Ail similar images must be similar in 
all features. 

• Weighted Minkowski function: All similar images are simi- 
lar in the same way (e.g., in the same set of features) [7]. 



We questioned the above assumptions upon observing how 
similar objects are located in the feature space. For this pur- 
pose, we carried out extensive data mining work on the 1.5M- 
iniagc datasct. To better discuss our findings, we introduce a 
term we have found useful in our data mining work. We defme 
the feature distance on the i*^ feature as ^ = |xi - yi|^i = 

In our mining work, we first tallied the feature distances be- 
tween similar images (denoted as d"*"), and also those between 
dissimilar images (denoted as 5~). Since we normalized fea- 
ture values to be between zero and one, the range of both 6'^ 
and 6~ are between zero and one. Figure 1 presents the dis- 
cribuiions of 6'^ and J". The x-axis shows the possible vahic 
of from zero to one. The y-axis (in logarithmic scale) shows 
the percentage of the features at differeni 6 values. 

The figure shows that 6"^ and S~ have different dislrfhution 
patterns. The distribution ofS'^ is much skewed toward small 
values (Figure 1(a)), whereas the dialribulion ot 6~ is more 
evenly distributed (Figure 1(b)). We can also see firom Fig- 
ure 1 (a) that a moderate portion of 6'^ is in (he high value range 
(> 0.5), which indicates diat similar images may be quite dis^ 
similar in many features. This observation suggests that the 
assumption of the Minkowski metric is inaccurate. Similar im- 
ages are not necessarily similar in all features. 

Next, we examined whether similar images resemble the 
query images in the same way. We tallied the feature distance 
(6'^) of the 144 features for different kinds of image transfor- 
mations. Figure 2 presents four representative transformations: 
GIF, cropped, rotated, and scaled. The x-axis of the figure de- 
picts the feature numbers, from 1 to 144. The first 108 feanires 
are various color features, and the last 36 are texture features. 
The figure shows that various similar hnages can resemble the 
query images in very different ways. GIF images have larger 
6"^ in color features (the first 108 features) than in texture fea- 
tures (the last 36 feamres). In contrast, cropped images have 
larger J"*" in texture features tlian in color features. For rotated 
images, the S"^ in colors comes close to zero, although its tex- 
ture feature distance is much greater. A similar pattern appears 
in the scaled and the rotated images. However, the magnitude 
of the S"^ of scaled images Ls very different from that of rotated 
images. 

We simimarizc our observations as follows: 

• Similar feature distoTice is distributed differently fi-om dis- 
ximilar feature distance. Similar feature distance skews to- 
ward small values, while dissim ilar feature distance shows 
more even distribution. 

• Similar images do not resemble the query images in all fea- 
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• IiTiagos ^simila^ to the query images can be similar in differ- 
ing fcalun:.s. For example, :fOme images resemble the query 
image ui texnire. others in color. 

The above obsemtions aot only refute the assunptions of 
Minkowski-iype distance functions, but also provide hints as 



Figure 2: The Average Feature Distances. 

to the Minkowski metric. When m < p, it counts only the 
smallest m feature distances between two objects, and the in- 
fluence of the (p - m) largest feaUire distances is eliminated. 
L>PF dynamically selects features to be considered tor different 
pairs of objects. This is achieved hy the introduction of ilmi 
which changes dynamically for difTerenl pairs of objects. In 
Section 3, we will show that DPF makes similar images ag- 



to how^a good distance ftinction would work^ Thefirai poinijs ^^^^^^^ ^ore compactly and locate closer to the query images, 

.1 ... ^..1 *ii r - siiQ^ii^aneously keeping the dissimilar images away from the 
query images. Tn other words, similar and dissimilar images 
are better s^^arared by DFF. 



3 Empirical Study 



that a distance function docs nut need to consider all features 
equally, since similar images may match only some features 
of the ciuery images. The second point is that a distance func- 
tion should weight features dynamically, since various Similar 
images may resemble the query image in differing ways. Tra- 
ditional relevance feedback methods 13J learn a set ''optimal** 
feaiurc weights for a query. For instance, if the user is more ^^piHcal study consists of two parts: training and testing, 
interested m color than m t«nire color feanires are weighted ^ J ^^^j 1.5M-image datasel to 

higher when simdw^^^^^^ What we have diseo^^^ ^^^^ ^^^^ ^j^^ ^ 

here js that this "static weightms is insufficient An cflbcUvc ^j^-image dataset to examine the cffeciiveness of DPR 
disunct; function must weight features differently when com- * 

paring the query image to different images. These points lead ^-1 Predicting m Through lyainJng 

to die design of thc^/iomfcpor/ja/ distance function. ... ^ . . 

We used the 60, 000 original images lo perform queues. Ap- 
2.2 Dynamic Partial Distance Function plying DPF of difierent m values to ihe 1.5M-image datasd, 

we tallied the distances frum these 60, 000 queries to their sim- 
Dased on die observations explained above, we designed a dis- ilar images, and their dissimilar images, respectively. Wc then 
ranee function to better represent the perceptual similarity. Let computed the average and the standard deviation of these dis- 



6% = » for i 1, - • • Wc first define sets Am <vs 
=» {The. srnalltsi rnS's oj (<5z, 

Thcn wc define the Dynamic Partial Distance Function 
(DPF) as 



rf(rn,r) = ( 6^^^ . 



(3) 



lances. We denote the average distance of the similar images to 
iheir queries as /i J» of the dissimilar images as /i^. We denote 
xhe standard deviation of the similar images' distances as 
of die dissimilar images as . 

Figure 3 depicts the elTcci of m (in the x-axis) on ^ij, < 
aj, aikd <7^. Figure 3(a) shows that as m becomes smaller, 
both /ij and /aJ decrease. The average distance of similar 
images (/ij), however, decreases at a faster pace than that of 
dissimilar images O^J )• instance, when we decrease m 



DPF has two adjustable parameters: m and r. Parame 
ler m can range from 1. to p. When m = p, it degenerates from 144 to 130, ftj decrca.ses from 1.0 to about 0.3, a 70% 
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Figui-e 3: The ElTeci of DPR 
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Figure 4: Search Performance of Different m air = 3. 



decrease, whereas /xj decreases from 3.2 to about 2.0, a 38% 
decrease. This gap uidicaies more sensitive to the m 
value than /ij . Figure 3(1)) shows that the standard deviations 
crj and (7j observe the some trend as ihc average distance.s. 
When vi decreases, similar images become more compaet iii 
the feature space at a faster pace than dissimilar images do. 
Our training itsull indicates that when vi is sec as IH, similar 
images are best clustered. 

3^ Testing New Distance Functions 

The test datasct consists of 1 00 similar-image sets, e^cJ) sei 
is con^oscd of 30 iniages. Of these 30 imiigcs, wc have the 
original unage» 24 transformed images (using the same trans- 
formation methods described in Section 2), and five images 
that are visually Ideniified as similar. We th^ added 50K ran- 
domly crawled Web images to these 100 x 30 images to form 
our test;et. 

We conducted 100 queries using the 100 original images. 
For eacli <juery, we recorded the ranks of its similar images. 
We expcrioienied wirfi m values from 84 to 144, with r fixed 
at three. Figure 4 depicts The experimental results. 

The precision-recall curves of selected rn values are plotted 
iu Figur>j 4(aj. The peak search performauce is achieved when 
m = 114, and it does significantly better than tlie Minkowski 
difstance {m = 144). Figure 4(b) plots the recall at selected m 
values tor (op-^: retrievals. As we dccrcasse the value of m from 
144, the recall improves steadily until m reaches 114, where 
the peak performance is achieved. Our DPF outperforms the 
Minkuwski disianee flincTion by 25 percentiles in recall. 

Becaasc of space limiiaiion, we present extensive exper- 
imental results and our comparison between the weigliied 
version of DPF and the weighted Minkowski metric in [4]. 
DPF coiisistenlly outperforms Minkowski-iype function sig* 



nificoDdy for finding smiilar images. 
4 Conclusion 

Tn this work we tackled one ftjndainental pzx)blem in im- 
age retrievals- how to measure perceptual simOarity between 
imagefv-^using data mining techniques. We discovered the dy- 
namic partial distance Junction (DPF) through mining a large 
set of visual data, and showed ihai DPF outperformed the ira- 
diiional functions by significant margins. The eirectivenesc of 
DPF can be explained by similarity theories in cognidve psy- 
chology [2, 4]. 
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